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Summary 


STEM Pathways aims to increase youths’ long-term interest, learning and achievement in 
STEM through a deliberate and interconnected system of STEM learning opportunities. 
STEM Pathways is a partnership of Minneapolis Public Schools (MPS), the Minnesota 
Department of Education (MDE), The Bakken, The Bell Museum of Natural History, 
Minnesota Zoo, STARBASE Minnesota, and The Works that tests a model for 
collaboration that could be expanded and replicated across more grade levels, schools, 
organizations, and communities. 


As part of a three-year project, STEM Pathways partner organizations began the first year 
in 2013-2014 with an intensive collaboration process to learn about each other’s program 
and make concept and theme connections, develop shared tools and strategies, and plan 
for project implementation. Partner organization refers to informal STEM education 
organizations, working in conjunction with the Minnesota Department Education and 
Minneapolis Public Schools. STEM Pathways began offering activities to fourth- and 
fifth-grade students at six MPS elementary schools: Bryn Mawr, Emerson, Jefferson, 
Keewaydin, Loring, and Pillsbury in 2014-2015. STEM Pathways continues to provide 
programs to fourth- and fifth-graders in those schools in 2015-2016. 


Evaluation 


A core component of STEM Pathways is the inclusion of an ongoing evaluation that can 
be used to inform programming. STEM Pathways participates in a rigorous, independent 
evaluation conducted by Wilder Research. Evaluators assess the implementation and 
outcomes of the project, using student survey and academic data and interviews with 
MPS leaders and STEM Pathways partner representatives. This report reflects evaluation 
findings in 2014-2015. 


Student survey results 


In fall 2014 and spring 2015, a survey was administered to fourth- and fifth-grade 
students at the six STEM Pathways schools in MPS. The survey assessed students’ 
STEM awareness, attitudes, interests, and activities. Changes in students’ responses in 
these areas from fall to spring may be associated with participation in STEM Pathways. 
A total of 705 students completed the survey in both the fall and spring for a response 
rate of 85 percent. Results are organized by areas that showed promising findings and 
areas that could be improved. 
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Promising results 


STEM interest and application 


Interest in engineering (“I like learning engineering”) increased overall from fall to 
spring. This increase occurred primarily among fourth-graders and appeared to be due to 
increased knowledge or understanding of what engineering is as “don’t know” responses 
from fourth-graders went down sharply from fall to spring while “agree a lot” responses 
went up very strongly. Perhaps related to this, agreement with the statement, “I think like 
an engineer to design solutions to problems,” increased among fourth-graders from fall to 
spring. The interest in engineering of all demographic groups examined (gender, eligibility 
for free or reduced-price lunch - FRL, English Language Learner - ELL status, and 
race/ethnicity —white students, students of color) increased from fall to spring as well. 


Interest in STEM as a whole (“I like learning STEM”) also increased among fourth- 
graders from fall to spring. This increase occurred especially among boys and ELL 
students. 


Overall, interest in other STEM subjects (math, science, and technology) did not change 
significantly from fall to spring. These results might still be viewed as favorable because 
over three-quarters of the students already liked learning these subjects at the time of the 
baseline survey in the fall. 


STEM relevance and awareness 


Students’ agreement that STEM knowledge is important to their futures increased from 
fall to spring. This increase occurred across grades and across almost all student demographic 
groups examined. 


Overall, students’ awareness of STEM (“I notice STEM in the world around me every 
day”) also increased from fall to spring. This increase primarily occurred among fourth- 
graders and among students of color. 


Knowledge of STEM careers 


Knowledge of STEM jobs (“I know about many jobs that use STEM”) increased from 
fall to spring in both fourth and fifth grades. This increase occurred in most of the student 
demographic groups examined. 
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Possible opportunities for growth 


Confidence in STEM abilities 


Confidence in STEM abilities (“I am really good at STEM”) decreased from fall to spring. 
Fourth-graders had a bigger decrease than fifth-graders, with more fourth-graders being less 
sure of their STEM abilities in the spring. Student demographic groups with decreases in 
their STEM confidence levels were girls, low-income students, ELL and non-ELL students, 
and students of color. 


Despite the decrease in agreement with the statement, “I am really good at STEM,” agreement 
with the statement, “I would be good at a job that uses STEM,” did not change significantly 
from fall to spring. Females, low-income students, and students of color tended to be less 
confident that they would be good at a STEM-related job compared to their demographic 
counterparts. 


Participation in STEM activities outside of school 


The proportion of students engaging in frequent STEM-related activities outside of school 
did not change significantly from fall to spring in either fourth or fifth grade. 


Interest in STEM careers 


Similarly, there was little change from fall to spring in students’ interest in having a job 
that uses STEM when they are older, with slightly over half agreeing with this item at both 
time points. Higher income students were more likely to have an interest in such a job. 


Student achievement results 


STEM Pathways long-term goal is to increase student academic achievement in STEM, 
especially in underrepresented groups (low-income students, racial/ethnic minorities, and 
females). Evaluators compared STEM Pathways students with similar peers attending 
other MPS schools on their spring 2015 Minnesota Comprehensive Assessments, Series 
Ill (MCA-III) math, reading, and science scores. A statistical method was used to 
“equalize” the two groups at baseline on factors that could influence achievement (i.e., 
student demographics and prior achievement). Results show no differences in academic 
performance between the treatment and comparison groups, with the exception of a slight 
advantage in reading for comparison group students in fourth grade. When differences in 
academic performance between treatment and comparison groups were examined within 
student demographic categories, no strong patterns of differences emerged. Differences 
were generally small within gender, income, ELL, and race/ethnicity categories. 


STEM Pathways Evaluation 3 Wilder Research, December 2015 


Results of Minneapolis Public Schools leader interviews 


Wilder interviewed MPS leaders to get their views on STEM Pathways effectiveness, 
alignment with Minnesota standards and MPS learning targets, collaboration, and benefits to 


students, teachers, and the district. Nine MPS leaders participated, including five principals of 


the six STEM Pathways schools. Leaders viewed their collaboration with STEM Pathways 
organizations positively. Several benefits to teachers, students, and the district were 
mentioned, including the following: 


m Provides students with good experiences in science and math 


= Expands student access to STEM learning activities, increases their enthusiasm and 
engagement in STEM learning, and promotes career exploration 


= Increases ability within schools to integrate science and literacy requirements 


= Provides modeling for teachers, inspires them, and increases their interest in STEM 
topics—giving them new ideas for ways to incorporate STEM education into their 
classroom 


= Connects and aligns informal education opportunities to state standards and MPS learning 


targets 


= Increases resources available to schools and the district and helps them use these resources 


more strategically 


= Strengthens relationships with the MPS district, including individual STEM Pathways 
schools and STEM Pathways partner organizations, resulting in better alignment of 
programming 


Results of STEM Pathways partner representative interviews 


Twelve representatives from the STEM Pathways partner organizations, including 
steering committee and implementation team members, were also interviewed by Wilder 
Research. Representatives were asked their views on the STEM Pathways collaborative, 
the program’s implementation and effectiveness, including impacts on their organization, 
program delivery, and staff development. 


Representatives of the STEM Pathways partner informal STEM education organizations 
indicated several successes from the first year of implementation including: 


= Full implementation of each informal STEM education partner’s programming for all 
the fourth- and fifth-grade classes at the six STEM Pathways schools 


= Enthusiasm and commitment on the part of all partner organizations 
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= Increased familiarity with, knowledge, and appreciation of what other partner 
organizations do and how their work complements the work of other informal STEM 
education organizations 


m Sharing of educational approaches between partner organizations, leading to new 
ideas and improved curriculum and instruction 


= Stronger alignment with and reinforcement of state standards and increased knowledge, 
on the part of the partner organizations, of MPS learning targets 


= Buy-in from leadership and teachers at MPS 


= Attention to and support for the Next Generation Science Standards 


Issues to consider 


Student survey results indicated fourth-grade students made gains in their STEM interest, 
awareness, and knowledge. Their gains tend to be higher than those of fifth-grade students. 
However, as the year progressed from fall to spring, their confidence in STEM abilities 
decreased. As of fall 2015, STEM Pathways partner organizations are considering ways 
to incorporate conversations with students into programming that could impact student 
confidence in STEM. Further, responsive interviews are planned for January and 
February 2016 in order to hear directly from students about how they perceive STEM. 


STEM Pathways’ potential effects on student academic achievement were examined 
using MCA tests, which are considered to be long-term indicators of potential effects of 
the program. Significant program effects may be unlikely in the short term, but could 
emerge after multiple years of program exposure. 


STEM Pathways partners and MPS leaders felt that STEM Pathways was successfully 
implemented, especially considering that it was the first year of implementation. Partners 
and leaders are hopeful for stronger project implementation over time. Both MPS leaders 
and STEM Pathways partners also pointed out some opportunities for project implementation 
efforts, including stronger and more consistent implementation of common project concepts, 
tools, and strategies across organizations; greater involvement of all partners; better 
communication and schedule coordination with the schools and the school district; 
involving teachers in STEM Pathways professional development meetings; and increasing 
funding for the project. 


Looking forward 


STEM Pathways partners continue to strengthen their partnership and deliver 
programming to fourth- and fifth-grade students in the same six MPS schools. Fourth- 
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grade students from 2014-2015 who are enrolled in one of the STEM Pathways schools 
in 2015-2016 as fifth-graders continue to receive STEM learning experiences from 
STARBASE and two new partners (Minnesota Zoo and Bell Museum). Another cohort of 
fourth-grade students in these schools also start participating in STEM Pathways, receiving 
programming from The Bakken, The Works Museum, and STARBASE. Wilder Research 
will continue to follow these cohorts of students in 2015-2016 and into later years, 
assessing the long-term impacts of the project, if funding permits. 


As STEM Pathways is well into the third year of the project, partners are focusing on 
strategic planning to determine the next phase of the project. Through this process, 
STEM Pathways partners will consider: 


= How to facilitate and support sustainability of access to a system of high-quality and 
interconnected informal STEM education for youth in collaboration with schools and 
districts 


= Articulating and strengthening of the local STEM learning ecosystem 


= Continuing to promote cross-organizational leadership to create and prioritize a 
culture of collaboration that builds authentic connections between organizations, 
people and programs; that articulates shared vision and goals; and utilizes shared 
measurement to evaluate progress toward goals 


= Continuing to support and even expand the network of informal STEM educators that 
STEM Pathways has created 


= Strategies for sustainability 
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Introduction 


STEM Pathways aims to increase youths’ long-term interest, learning and achievement in 
STEM through a deliberate and interconnected system of STEM learning opportunities. 
STEM Pathways is a partnership of Minneapolis Public Schools (MPS), the Minnesota 
Department of Education (MDE), The Bakken, The Bell Museum of Natural History, 
Minnesota Zoo, STARBASE Minnesota, and The Works that tests a model for 
collaboration that could be expanded and replicated across more grade levels, schools, 
organizations, and communities. 


The partnership was formed in spring 2013 with the following objectives: 


= To provide deliberate, coherent, and connected pathways of meaningful in-school and 
out-of-school STEM learning experiences that contribute to a local STEM learning 
ecosystem and lead to the achievement of shared youth outcomes 


= To test anew model and culture for how informal STEM education organizations 
work together, emphasizing collaboration, shared vision, goals, knowledge, strategies, 
and measurements 


During the first year of the project (2013-2014), STEM Pathways partner organizations 
developed and used a collaboration process to learn about each other’s organizations and 
programs; explore important topics in formal and informal STEM education fields; 
determine the overarching themes and concepts connecting the STEM Pathways partner 
programs; and develop common tools, strategies, and messages that would help students 
experience long-term and interconnected STEM learning and build their confidence 
(Figure 1). The collaborative efforts of sharing goals, strategies, and tools among partner 
organizations to build interconnected learning experiences for students is a major shift in 
the culture of informal STEM education organizations that traditionally work in isolation 
from one another. Partners met regularly through a series of working meetings. A 
summary of the working meeting topics is described on pages 10-12. 
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STEM Pathways collaboration process 


Building knowledge about: 


Content themes, instructional practices, 
learning goals of partner programs via 
observation and lesson comparisons 
School district learning targets associated 
with state standards 

Next Generation Science Standards science 
and engineering practices 

Learning ecosystems 


Source: STEM Pathways 
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leads to dicovery of... 


Shared values and 


connecting themes: 


Engineering Design Processes (EDPs) (4th 
grade) 

Systems and impact of change (5th grade) 
STEM careers (both grades) 

Everyone can do STEM (both grades) 


made visible through... 


Instructional tools and strategies: 


Shared vocabulary for key STEM practices 
Linkages between different EDPs 

Visual tools for learning about systems and 
impact of change 

Interactive career exploration tool, 
discussions of STEM careers and broad value 
of STEM skills 

Instructional strategies that include and 
encourage all students and build confidence 
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STEM Pathways worked with the Minneapolis Public Schools’ Teaching and Learning 
director and STEM coordinators to select the schools and grade levels where informal 
STEM learning opportunities were most needed. School principals and teachers received 
extensive information about and expressed their willingness to participate in STEM 
Pathways prior to launching the project. 


In the second year (2014-2015), STEM Pathways began offering activities to fourth- and 
fifth-grade students at six elementary schools: Bryn Mawr, Emerson, Jefferson, 
Keewaydin, Loring, and Pillsbury. 


Wilder Research serves as the independent evaluator of STEM Pathways. As such, 
Wilder Research worked closely with the STEM Pathways partners to design the 
evaluation and with MPS to implement the evaluation activities. 


Funding 


STEM Pathways received funding from the Department of Defense (DoD) STARBASE, 
Boston Scientific and in-kind contributions from the partner organizations, Minneapolis 
Public Schools, and Minnesota Department of Education. Funding from DoD STARBASE 
and Boston Scientific supported the evaluation of the project and the project director 
position. Recently, STEM Pathways also received additional funding from Pentair. 
Partner organizations dedicated substantial amounts of staff time to the implementation 
efforts and subsidized the program fees. 


Content of the report 


This evaluation report focuses on first-year implementation of STEM Pathways and 
short-term project outcomes during the 2014-15 school year. 


The report is organized into the following sections: 
= Partnership efforts 

m= Research methods 

= Implementation 


= Outcomes 


Looking forward 
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Partnership efforts 


Planning year (2013-2014) 


The STEM Pathways steering committee which consists of key representatives from 
informal STEM organizations, MDE, and MPS met regularly throughout the planning 
year. Additionally, working meetings were held for STEM Pathways educators and 
steering committee members to learn about each other’s organization and make connections. 
Implementation team and grade-level workgroups were created to plan and coordinate the 
activities for fourth- and fifth-grade students. 


Below is a short summary of four working meeting topics in 2013-2014 (note that there 
were multiple topics covered at each of the meetings). These meetings were each held at 
four different partner organizations and each attended by about 25 STEM educators and 
leaders and MPS leaders and teachers. 


= Engineering Design Process. STEM Pathways educators and leaders and STEM 
educators from Minneapolis Public Schools explored the various Engineering Design 
Processes (EDPs) used by the partner organizations, identified the common concepts 
and language across the EDPs and opportunities to make connections between them. 


m Next Generation Science Standards (NGSS). In this professional development 
session, the MDE STEM Specialist and the MPS Elementary STEM Coordinator 
engaged partners in lessons designed to focus on developing science and engineering 
practices. 


= STEM Pathways logo development. Sample logos were developed by STEM educators 
and meeting attendees voted on the final logo. Attendees discussed how the logo can 
be used to brand STEM Pathways and be used by each partner organization and how 
the logo represents the messages STEM Pathways wants to convey to teachers, 
students, and families. 


= Learning targets in the MPS science curriculum. The MPS Elementary STEM 
Coordinator led discussions about how MPS uses student-centered learning targets 
with individual science and engineering lessons to build student mastery of science 
standards. STEM Pathways partners discussed how the same approach is being used 
or could be used in their programming. 


= Writing in science. STEM Pathways partners explored research-based practices for 
engaging students in writing in meaningful ways for practicing and developing 
science and engineering skills. 
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First implementation year (2014-2015) 


In 2014-2015, the STEM Pathways steering committee and educators continued to meet 
and refine the project efforts. A summary of working meeting topics and task force 
efforts in the second year of STEM Pathways (first implementation year) follows. 


Development of STEM Pathways Portfolio. The Portfolio is designed as a tool to help 
students reflect on their STEM experiences, showcase their STEM-related activities, 
and track their progress in STEM. It consists of a STEM Passport, a place to record 
participation in each of the STEM Pathways programs; reflection pages, where students 
can record information and thoughts after STEM partner experiences; additional 
STEM activities that can be completed in class or at home; the Game of STEM, 
where classroom teachers recognize students’ progress and accomplishments in 
STEM that allows students to “level up” in STEM. The Portfolio is designed for use 
by teachers and students. STEM Pathways educators are encouraged to reference the 
Portfolio and how it can be used during their programming. 


Development of STEM Pathways Career Interactive. Students can explore a wide 
range of STEM careers designed to build off the momentum of the partner 
experiences back at school. For example, for careers related to their experiences with 
Minnesota Zoo and the Bell Museum, students can view videos of an entomologist, a 
neurobiologist, or a frog scientist describing their work. For careers related to their 
experiences at The Bakken, STARBASE, or the Works, students can view videos of a 
solar car engineer, a wind researcher, a renewable energy engineer or a robot engineer. 
The videos are produced by Twin Cities Public Television, Dragon Fly TV. 


MPS learning targets follow-up. STEM educators practiced developing learning 
targets specific to their programs and explored how each program’s unique set of 
learning targets, in combination with ones specific to MPS curriculum, support 
student mastery of standards. 


STEM learning ecosystems. STEM educators explored the concept of natural 
ecosystems, and possible changes and implications of those changes to the 
ecosystems, and then applied these concepts to imagine a local STEM learming 
ecosystem including its components and resources to thrive. 


Implementation team best practice sharing: STEM integration, storytelling, and live 
animals. Instructors from three partner organizations shared key practices from their 
programs. STARBASE Minnesota presented their approach to integrating the STEM 
subjects (science, technology, engineering and math) and how this informs the 
curriculum review process that they use to make program improvements. The Bakken 
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shared storytelling techniques that they use to engage students in invention and scientific 
discovery. The Minnesota Zoo shared their approach to using live animals in their 
programs and how they use this opportunity to inspire empathy for and connection to 
the natural world. Following these presentations, attendees discussed what they 
learned from each other and points of connection between STEM Pathways programs. 


Defining STEM Pathways values and key messages. Implementation team and 
steering committee members gathered to revisit core assumptions of STEM Pathways 
collaboration and the values that drive its work. Partners affirmed values related to 
educational experiences that: integrate science, technology, engineering and math, 
rather than treat them as isolated topics; promote the value of STEM education for all, 
not just those who will enter the traditional STEM workforce; help students develop 
positive STEM confidence and competency beliefs; and are interconnected and 
reinforcing so that students take part in a web of STEM experiences that share 
concepts and themes over time. 


Additionally, STEM Pathways partner representatives presented the work of STEM 
Pathways at the following professional development conferences: 


sth Annual Minnesota STEM Network Conference, April 2014* 

University of Minnesota Pre-K-12 Network Conference, February 2015 

National Science Teachers Association (NSTA) National Conference, March 2015 
Minnesota STEM Network and Ignite Afterschool Joint Conference, April 2015 


This presentation happened in the first (planning) year of the project 
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Research methods 


In consultation with STEM Pathways steering committee, Wilder Research developed an 
evaluation plan for the first project implementation year with the potential for extending 
it as funding permits. Research methods are described below. 


Research design 


Wilder Research’s evaluation has implementation and outcome components addressing 
the following goals: 


«= Implementation evaluation. Assess project implementation and provide feedback 
useful in strengthening implementation efforts. 


= Outcome evaluation. Assess short-term outcomes for project activities and provide 
initial reporting on long-term indicators reflective of major project goals. 


Research questions 


First-year evaluation efforts were guided by the following research questions, organized 
into the implementation and outcome components of the evaluation: 


Implementation questions 


The implementation component of the evaluation is an important mechanism for 
understanding project efforts that seem effective and for providing feedback useful in 
strengthening project activities. Key research questions associated with the implementation 
evaluation include the following: 


1. How successfully is the STEM Pathways model being implemented? 


2. What partner and school characteristics are associated with strong 
implementation? In what ways can implementation be strengthened? 


3. How well does the collaboration function, and how can it be strengthened? 
4. How effective is professional development, and what are its future needs? 


5. What are the core components of the program model and conditions for 
replication? 
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Outcome questions 


The outcome component of the evaluation addresses changes in systems, perceptions, and 
student outcomes related to STEM. During the first evaluation year, outcome measurement 
focuses on initial system changes, short-term outcomes associated with project activities, and 
initial measurement of student outcomes reflective of long-term project impact. Research 
questions for this component of the evaluation include the following: 


6. Does the STEM Pathways model enhance the short- and long-term outcomes of 
populations underrepresented in STEM? 


7. How well does the model work for specific underserved populations? 
8. What impacts does the model have on informal STEM education organizations? 
9. What impacts does the model have on classroom teachers? 


10. What are the implications for the field of informal science education? 


Data collection procedures 


Data collection procedures for the implementation and outcome components of the 
evaluation are described below. Specific research questions addressed through each 
method are denoted. 


STEM Pathways partner interviews 
(Questions 1-5, 8, 10) 


In spring 2015, Wilder Research conducted one-to-one phone interviews with STEM 
Pathways partners regarding the goals, implementation, and accomplishments of STEM 
Pathways and about suggestions for changes or improvements for STEM Pathways in the 
future. Interviews were conducted with members of the steering committee and members of 
the implementation team. Respondents included the project director for STEM Pathways as 
well as three representatives from STARBASE; two representatives each from The Bakken 
Museum, Minnesota Zoo, and Minneapolis Public Schools; and one representative each 
from The Works Museum and The Bell Museum of Natural History. 


Minneapolis Public Schools leader interviews 
(Questions 1-4, 6, 7, 9, and 10) 


Wilder Research also conducted one-to-one phone interviews with MPS leaders in spring 
2015, including five of the six school principals, GEMS & GISE coordinator, STEM 
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curriculum integration specialist, the director of elementary education, and the director of 
Teaching and Learning. The survey asked MPS leaders about the implementation and 
accomplishments of STEM Pathways as well as suggestions for changes or improvements 
for STEM Pathways in the future. 


Student pre-post survey 
(Questions 1, 6, and 7) 


In fall 2014 and spring 2015, a survey was administered to fourth- and fifth-grade 
students at the six STEM Pathways schools in MPS. The survey was administered to 
students as a group in their classrooms by Wilder Research staff in both fall (September 
2014) and spring (May-June 2015). After a brief explanation of the survey, Wilder 
Research staff read the questions and students provided their answers on paper-and-pencil 
survey forms. A few students who were absent on the day the survey was administered 
completed it later. Students’ parents or guardians were informed about the survey by 
letter and could have their child excluded from the survey if they wished by contacting 
the school. 


Student survey results are reported for students who completed all or most of the survey in 
fall and spring in the STEM Pathways schools. Of 829 eligible fourth- and fifth-graders from 
the six STEM Pathways schools, 705 completed all or most of the survey in both the fall 
and spring for a response rate of 85 percent. 


Academic achievement data 
(Questions 6 and 7) 


This first year evaluation report provides student achievement results on indicators 
reflective of long-term project goals. Data are presented in the areas of math, science, and 
also reading, in recognition of the potential broader impact of STEM on student development. 
It is important to recognize that moving the needle on these indicators takes time, and 
numerous factors contribute to that effort. However, monitoring these indicators during and 
beyond the initial project period is important in that ultimately, this project is intended to 
assist with systems change efforts that increase MPS students’ success in STEM areas and 
narrow achievement gaps. Data are provided from the Minnesota Comprehensive 
Assessments, Series III (MCA-III), in math, science, and reading. 


School attendance will also be included as a student outcome measure in the evaluation in 
future years. It seemed too early to include it as an outcome measure during the first year 
of project implementation. This is because attendance was measured throughout the year 
as the project was being implemented, and consequently, it is not an end-of-year measure 
that would have a greater chance of being impacted by project programming. 
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Data analysis 


Minneapolis Public Schools leader interviews and STEM Pathways partner 
interviews 


Evaluators reviewed the interview transcripts and organized the respondent comments by 
key themes. As appropriate, these comments were also organized by the position or role 
of the respondents in their organization or project. 


Student survey 


Data are analyzed for the same 16 closed-ended survey items included in the fall and 
spring student survey, permitting analysis of changes in students’ responses to the items 
from the beginning to the end of the 2014-15 school year. Of the 705 students who 
completed all or most of the survey in both the fall and spring, 353 were fourth graders 
and 352 were fifth graders. Response options to the survey items were: “agree a lot”, 


“mostly agree”, “agree a little’, “don’t agree”, and “don’t know”. Those who agreed a lot 
or agreed mostly were considered to be in agreement with an item. 


Statistical tests (McNemar Test, two-sided) were conducted to determine whether change 
from fall to spring in the percentage of students responding “agree a lot” or “mostly agree” to 
each item was statistically significant (p<.05). When the terms “significant” and “not 
significant” are used in describing such changes in responses, these terms are referring to the 
results of the statistical tests. 


Differences in survey results were examined by grade and by student demographic 
characteristics: gender, eligibility for free- or reduced-price lunch (an indicator of students’ 
family income), ELL status, and race/ethnicity (white students and students of color). 


Student academic achievement 


To assess student achievement, the evaluation uses a quasi-experimental design and 
analyzes data for cohorts of fourth-grade and fifth-grade students in six STEM Pathways 
schools (treatment) and similar students in non-STEM Pathways schools (comparison). 
All schools are in the Minneapolis school district. 


The six “treatment” schools participating in STEM Pathways include four K-5 schools, 
one K-8 school, and one 3-8 school. The schools are: 


= K-5 schools: Bryn Mawr, Emerson, Loring, and Pillsbury 
= K-8 school: Jefferson 


m 3-8 school: Keewaydin 
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The evaluation includes cohorts of fourth- and fifth-grade students in these schools in 
2014-2015. Another cohort of fourth-grade students will be added in 2015-16 and the 
evaluation will continue to follow these students over time, as funding permits. Figure 1 
shows the study cohort groups. 


1. STEM Pathways cohort groups 


2014-2015 2015-2016 2016-2017 
Cohort 1 4" grade 5% grade 6" grade 
Cohort 2 5" grade 6" grade 7" grade 
Cohort 3 =e: 4‘ grade 5'* grade 


Note: STEM Pathways was implemented in 2014-2015. Evaluation can potentially follow the students who ‘graduate’ from 
Pathways into the later grades (6%, 7 and 8" grades and high school), assessing their long-term academic outcomes. 


School performance of the STEM Pathways cohort students (treatment) were compared 
with those of same-grade students from non-Pathways schools (comparison). The evaluation 
uses a Statistical technique (propensity score analysis) to “equate” the treatment and 
comparison group students at baseline, so that the potential effects of STEM Pathways 
treatment on student outcomes are not confounded by differences in student characteristics 
between the groups that could influence student achievement. These characteristics 
include students’ prior academic achievement (third- and fourth-grade MCA-III math 
scores for fourth- and fifth-grade students, respectively) and several student demographic 
characteristics (free or reduced-price lunch eligibility, ELL status, special education 
status, gender, and race/ethnicity). 


Potential STEM Pathways effects were examined through analysis of differences between 
the treatment and comparison groups on student achievement measures: MCA math, 
reading, and science scores. Differences in student achievement between the treatment 
and comparison groups were examined overall for fourth- and fifth-graders, and within 
demographic subgroups. A detailed description of the statistical techniques used in 
analyzing the student academic data is presented in another report, STEM Pathways 
Student Academic Achievement Results for the 2014-15 School Year (Mueller, 2015). 


Overall strengths and limitations of methodology 


Strengths 


The study data were collected from multiple perspectives, including STEM Pathways 
partners, MPS leaders, and students. Survey and interview instruments were developed 
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thoughtfully and collaboratively with the STEM Pathways steering committee and 
allowed for triangulation of respondent perspectives on several topics (e.g., partners and 
MPS leaders identified the status of collaboration). Overall, response rates were high, 
especially on the student survey. Additionally, student demographics and academic 
achievement data for treatment and comparison students were provided to the study by 
MPS Research, Evaluation, and Assessment department (REA). 


Limitations 


Results of student surveys (i.e., changes in their responses from fall to spring related to 
STEM learning, interests, and activities) may be associated with participation in STEM 
Pathways. However, caution is needed in attributing them to STEM Pathways because other 
STEM experiences students may have had in and out of school during the same period could 
have contributed to the changes as well. Additionally, comparisons to perspectives of non- 
participating students were not available as it was not feasible to conduct the student survey 
in a comparison group. 


A classroom teacher survey was not collected during 2014-2015. However, the STEM 
Pathways project coordinator and STEM Pathways staff met with classroom teachers at 
the end of the school year to get their feedback about the project. Information provided 
by the teachers was used to help in planning for the second year of project implementation. In 
2015-2016, a classroom teacher survey will be conducted by the evaluator. 


Minnesota Comprehensive Assessments (MCA-III) were used to assess student academic 
outcomes in math, science, and reading proficiency. MCA tests are the academic 
achievement measures administered to students statewide each spring, and serve as 
accountability measurement tools in Minnesota for the federal No Child Left Behind law. 
We consider MCA tests to be long-term indicators of potential effects of the program. 
Measurement for short-term outcomes to assess student comprehension on specific 
subtopics/units of STEM was not available. 


Data privacy and research consent 


Wilder Research worked with the MPS REA to ensure evaluation procedures were 
consistent with district data privacy and research consent requirements. Wilder Research 
and MPS developed a data confidentiality agreement at evaluation onset safeguarding data 
security, and evaluation plans were approved by REA. Data collection instruments and 
consent procedures used in the course of this evaluation were also reviewed by REA prior 
to their implementation. 
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Implementation 


In 2014-2015, STEM Pathways’ goals for implementation include: 


= Giving students access to multiple in and out-of-school STEM Pathways partner 
programs 


= Offering connected experiences across partner programs 


= Offering high quality, relevant STEM experiences for Pathways students 


Research question 1. 
How successfully is the STEM Pathways model being implemented? 


During the 2014-2015, fourth- and fifth-grade students received almost 30 programming 
hours from STEM Pathways partners. Following are the grade-level program 
descriptions. 


Programs for students 


Fourth grade programs 


In the fourth grade STEM Pathways program sequence, students participate in real world 
STEM learning that helps them see a bigger picture of how STEM really works and how 
it relates to them, specifically around the practices and process of science and engineering 
and STEM careers. Fourth grade programs are offered by the following organizations. 


The Works Museum 
Program hours: 3-hour fieldtrip 


Students travel to the museum to explore interactive museum exhibits and participate in 
the Maze Engineering Workshop. Students use the Engineering Design Process to design 
and construct their own maze, pinball or pachinko game to take home. They experiment 
with changes in speed and direction and the effects of gravity and friction. 


The Bakken Museum 
Program hours: 1 assembly hour, 2 classroom hours and a 3.25-hour fieldtrip 


On the first visit to the school, Bakken educators invite all fourth-graders in the school to 
explore what it means to “Wonder, Try, Discover, and Share” — important habits of every 
scientist and engineer. A key goal of this presentation is for all students to see themselves as 
scientists and engineers. On the second visit, a Bakken educator works with fourth-grade 
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students in each classroom to guide them in applying “Wonder, Try, Discover, and Share” as 
they re-invent the battery. As scientists and engineers, students practice supporting their 
claims with evidence and learn the importance of taking risks and making mistakes in the 
process of discovery. 


Following the assembly and the classroom visit, students travel to The Bakken Museum 
to participate in a hands-on workshop and a guided exhibit tour. In the Magnets and 
Electromagnets workshop, students discover properties of magnets by recreating historical 
experiments and explore magnetic force and field and the relationship between magnetism 
and electricity. During the guided tour, students explore the history of science and invention 
via storytelling and interactive exhibits. Every student takes home materials to construct an 
electromagnet for their own experiments. 


The final classroom experience is led by a Bakken educator and a volunteer from the STEM 
workforce who help students connect the STEM they do in school to real careers. Students 
engage in problem solving to explore a real world challenge involving electromagnetism and 
energy. 


STARBASE Minnesota 
Program hours: 20 hours 


At STARBASE Minnesota, students learn to be engineers as they design solutions to 
challenges related to traveling from Earth to Space. As students design rockets, rover 
programs, landers, and other prototypes, including 3D printed rocket fins, they develop 
and utilize science knowledge and skills related to forming questions, conducting 
experiments, inertia, heat transfer, properties of air, and more. Students apply math skills 
such as data collection, median, measurement, estimation, and graphing as tools in the 
engineering process. 


Fifth grade programs 


The fifth grade STEM Pathways program sequence builds on the fourth grade sequence 
and introduces additional connecting themes. STEM Pathways fifth grade programs 
address the complexity of and impact of change on systems. Fifth grade programs are 
offered by the following organizations. 


Minnesota Zoo 


Program hours: 4-hour field trip 


Through the Minnesota Zoo’s Zoo Safari program, students learn about animals and their 
habitats and participate in the Bare Necessities class. In this class, students learn about 
how ecosystems function and make predictions of impacts on the habitats as a result of 
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changes in the ecosystems. Students gain deeper understanding of ecosystem dynamics 
and greater appreciation for the important roles everyone can play in the system. 


Bell Museum of Natural History 
Program hours: 1 classroom hour and a 4-hour field trip 


The fifth grade curriculum (a core MPS program), Honey Bees, Pollinators, and Food, 
examines human interdependence with nature. Student scientists exercise critical thinking 
and literacy skills as they explore the concept of biological diversity, the process of science, 
and the connection between our food and a healthy environment. Students use scientific tools 
and processes, such as the microscopes, forceps, and dissections to make observations of the 
anatomy of bees and flowers. Students learn about the current research at the University of 
Minnesota and apply their knowledge about pollinators as they plan meals, and discuss and 
analyze the impact of the disappearance of pollinators on the meals they’ve prepared. A tour 
of the Bell Museum’s dioramas allows the students to see some of the native habitats that are 
beneficial to pollinators. This program is structured to empower student scientists to learn 
more about where their food comes from and how humans are connected to the many 
processes and relationships in obtaining food and maintaining a healthy environment. 


STARBASE Minnesota 
Program hours: 20 hours 


At STARBASE Minnesota, student engineers develop and utilize their knowledge of 
energy transfer and energy systems, properties of air, Newton’s Laws of Motion, and 
more as they design a mission from launch, to landing, to living on Mars, including 
designing and testing 3D printed prototypes. Students apply math skills related to 
coordinate graphing, calculating mean, volume, area, and graphical data analysis as they 
complete their mission. 


MPS program 


In addition to the above programs, students can choose to attend the MPS STEM 
program. 


GEMSIGISE 
Afterschool and summer program 


Girls in Engineering, Mathematics and Science (GEMS) and Guys In Science and 
Engineering (GISE) are elective afterschool and summer programs designed specifically 
for fourth-through eighth-grade students in all MPS schools. Project- and problem-based 
learning experiences promote integration of engineering and technology concepts, 
content, and process. Students participate in an academic culminating event involving a 
design challenge. 
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In addition to connecting themes, STEM Pathways programs use shared instructional 
tools and messaging, which include vocabulary cards that help students and teachers 
recognize specific content connections between programs. For example, “conflict cards” 
are used to help the fifth-grade cohort students understand the impacts of a change on 
systems. After introducing the concept of systems, these conflict cards challenge students 
to determine what would happen when a conflict occurs in a system. STEM Pathways 
also provides tools that teachers and students can use at school or home, such as the 
STEM Pathways Career Interactive mentioned earlier. 


@) natural system |(j)designed system Vocabulary Cards - Common vocabulary is utilized by partner organizations, including vocabulary cards 


Conflict Cards - Used in the fifth-grade programs, each conflict card introduces a 
change to a system that students have been studying. Students are then challenged to 
determine what would happen when that change occurs. 


Student Portfolio - All students receive a STEM Pathways portfolio 
to track their STEM Pathways, classroom, and out-of-school STEM 
experiences throughout the school year. Reflection questions and 
pages to track interesting STEM careers are also included. 


Game of STEM - A game board included in student 
portfolios and a classroom poster used by teachers to recognize 


each student's STEM learning. Students “level up” as they do 
more and more STEM. 


Source: STEM Pathways 
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Perspectives from partners, MPS leaders, and students 


The following section provides results from interviews with STEM partner organizations 
and MPS Leaders. Interviewee responses are organized by research questions. Detailed 
interview results are also presented in two separate reports: Results from Spring 2015 
Interviews with STEM Pathways Partners (Streich and Bartholomay, 2015) and Results 
from Spring 2015 Interviews with Minneapolis Public Schools Leaders (Streich and 
Bartholomay, 2015). Results of an open-ended question in the spring student survey are 
also included. 


Overall, STEM Pathways partners and MPS leaders felt that STEM Pathways was 
successfully implemented, with all the partners implementing a total of almost 30 
programming hours at the six STEM Pathways schools. Although still at an early stage of 
implementation, partners and leaders mentioned seeing the connections among partners’ 
work in using common vocabulary, crosscutting concepts, practices, and core ideas. 
Partners and MPS leaders mentioned that STEM Pathways programming aligns with the 
state standards and increasingly supports the district curriculum. MPS leaders mentioned 
the benefits of the project for the students’ STEM learning. Comments from partners and 
MPS leaders follow. 


Successes 


A major accomplishment of the first operational year was the implementation of each 
partner’s programming for all the fourth and fifth grade classes at the six Pathways 
schools 


Just the fact that, in its first year of trying to provide 30 hours of programming in fourth and 
fifth grade at six schools, that to a large degree it happened, | think is a big success. 
— Steering committee member, STEM Pathways partner interviews 


Connections among partners’ work and with MPS 


Finding: Most partner organizations refer to other partner organizations when working 
with students. 


I very intentionally will reference STARBASE and the MN Zoo, what their programs are 
about, the scientific process that they have participated in and how that flows through all 
three of our programs. — Implementation team member, STEM Pathways partner 
interviews 
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Finding: Relationships between the MPS district, including individual Pathways schools, 
and STEM Pathways partner organizations have been strengthened resulting in better 
alignment of programming. 


...1 definitely think that the organizations working together and with the district is a huge 
success. | think [the] communication, sharing of goals and ideas, and education is a huge 
success. | think having common language, vocabulary and having those connections 
between organizations so that students can pick up on that is a really big deal. - MPS 
leader interviews 


Finding: Partners began to use common vocabulary, crosscutting concepts, practices, and 
core ideas. 


The idea of systems and interdependence in systems, as well as impact of change on 
systems, is the big crosscutting concept that we're trying to have woven through this 
whole fifth grade experience. A lot of that is introduced by us. | don’t know that we did a 
super good job this year, but | think next year, now that we've been through this once, my 
instructors, when they introduce this kind of stuff, will be able to say - you're going to see 
more systems or think about when you encounter systems and will know what to talk to 
them about moving forward. We've talked a lot about it and are working on how to do a 
good job of pulling that stuff through. 

— Implementation team member, STEM Pathways partner interviews 


In the spring survey, students were asked to name one experience or learning that was 
similar between two different STEM Pathways programs. Students described many 
similarities in their STEM learning experiences across STEM Pathways programs. Their 
most frequent answers included learning about science, STEM in general, and about 
animals or nature. They also mentioned other learning not related to STEM, such as 
learning is fun and they can build anything (Figure A3). 


Alignment with state standards, NGSS, and supports of MPS curriculum 


Finding: The partners felt that STEM Pathways programming from each site is strongly 
aligned with state standards and expressed improved knowledge of and alignment with 
MPS learning targets. 


= The partner organizations were working to align their programming with the state 
standards before their involvement with STEM Pathways, and have now become 
more familiar with how the district defines them and how they are implemented by 
teachers within the district. Working with each other, with the district, and with teachers 
helped the informal education organizations to make even greater connections to the 
standards. 
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... ight now at [Partner Organization] all of our programs are aligned to the state 
standards, but we hadn't done any work to specifically align them to the learning targets in 
Minneapolis. That was something that we started to look at this year. — Steering 
committee member, STEM Pathways partner interviews 


Finding: MPS leaders felt that STEM Pathways supports some aspect of the MPS 
curriculum. 


[STEM Pathways partners] know what those standards are for the fifth grade, and when 
our students go on a field trip [at the partner site] they will highlight those things or they'll 
make sure they're using some of the same vocabulary the teacher might be using in labs 
or lessons here. — MPS leader interviews 


Talking with the teachers, they feel it really supports the MPS curriculum 100%. - MPS 
leader interviews 


Our curriculum is aligned to the state standards that teachers are charged with teaching 
each year. Insofar as the things I've looked at, some of the written materials and what | 

saw at STARBASE, there’s also an alignment there with those same standards. - MPS 

leader interviews 


Finding: The partners have kept in mind the Next Generation Science Standards (NGSS) 
as they have been developing their programming. It is understood that these standards are 
important and may be adopted by Minnesota in the near future. 


As soon as we started going down the road in developing the curriculum and talking about 
how we were going to present the material, we were already looking at the NGSS at the 
same time. — Implementation team member, STEM Pathways partner interviews 


Connections with students’ STEM learning at school 


Finding: There is growing enthusiasm and engagement in STEM learning from students 
as aresult of STEM Pathways field trips. Students are making connections between their 
STEM Pathways experiences and what they are learning in school. 


What our students have shared is that they think being a scientist is cool, being a scientist 
is fun, and building stuff is fun. | remember distinctly sitting with a few students saying ‘I 
need to learn this, | need to learn this because this is what we practiced when we were at 
STARBASE. We practiced this in math, so | need to figure this out.’ So they’re making 
these connections and to me that’s what it is. If a student walks away learning something 
and seeing the value and its purpose and how it connects to their future, it was a 
successful day. — MPS leader interviews 


Opportunities for improvement 


MPS leaders and partners mentioned some challenges they experienced in the first year 
of the project. These challenges included partners are at different points in their 
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understanding and implementation of the model, lack of communication or coordination 
with schools, and scheduling issues. At the same time, interviewees felt hopeful that the 
project is making improvement on these areas. 


Understanding of the model 


Finding: Partners varied in their understanding and implementation of shared 
vocabulary, crosscutting concepts, practices and core ideas. 


| feel like we talked about it a lot, but | don’t know that we ever came to any consensus 
about what that shared vocabulary was going to be. So | don't feel like that’s done. It's 
sort of a work in progress, but | do feel like it's important. -Steering committee member, 
STEM Pathways partner interviews 


I'm still a little vague on what that means... We have core ideas that we're trying to 
emphasize. Whether they're crosscutting concepts I’m not exactly sure. — Steering 
committee member, STEM Pathways partner interviews 


Finding: The STEM Pathways Portfolio was developed for classroom teachers to use in 
the classroom to support student STEM learning. Some partners said that teachers have 
given positive feedback on the Portfolio as a useful tool. Referencing to the STEM 
Pathways Portfolio varied across organizations, with some organizations referencing it 
more regularly than others. Similarly, discussion of careers or promoting the Career 
Interactive website varied across partner organizations. 


! have absolutely no idea how it worked...!’ve heard, just through meetings, talking with 
[instructor] who’s the main liaison between the teachers and the [informal educators], that 
they liked it and it was useful, but | haven't spent much time thinking about it. - 
Implementation team member, STEM Pathways partner interviews 


...but we don’t have a ton of time to focus on career stuff. As | mentioned before we only 
have 45 minutes of hands-on time with the kids. — Implementation team member, STEM 
Pathways partner interviews 


Communication or coordination 


Finding: Communication between teachers and informal education sites prior to the field 
trip visits varied for each partner organization. Often teachers didn’t know what they 
were going into when they went on a STEM Pathways field trip. However, several 
principals felt that, after this year, teachers will be able to plan better to fit the learning 
from each site into their curriculum and learning sequence. 


The first year was a little tricky with teachers for navigating. Now they can see for next 
year how they will connect it. Connecting has happened after, next year [it] could happen 
more before. - MPS leader interviews 
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Teachers weren't sure about what would be happening at the Bell Museum coming up. — 
MPS leader interviews 


Finding: Scheduling between the schools and the partner organizations presented 
logistical challenges for some organizations. 


That can be pretty tricky because we weren't able to schedule until later in our scheduling 
process. Picking dates that are available for teachers and their classes was a big 
challenge. — Implementation team member, STEM Pathways partner interviews 


Research question 2. 


What partner and school characteristics are associated with strong implementation? In what 
ways can implementation be strengthened? 


Drawing from their experiences during the first year of implementation, both STEM 
Pathways partners and MPS leaders felt more time for planning and reaching consensus 
on what they want to accomplish is important for successful implementation. Partners 
and MPS leaders also indicated that scheduling activities and coordinating with teachers 
or schools earlier and communicating costs, availability of funding, and what each 
organization is able to contribute are needed. Involving teachers and parents was mentioned 
by MPS leaders and more involvement from steering committee was mentioned by 
partners. MPS leaders suggested getting feedback from teachers about the project. 
Although a teacher survey was not collected in 2014-2015, the project coordinator and 
STEM Pathways staff met with some of the teachers informally to get their feedback 
about what worked well and what needed enhancement or improvement. 


More time for planning and reaching consensus on goals 


m= Partners recommended agreeing on shared vocabulary, crosscutting concepts, 
practices, and core ideas to strategically implement across organizations. They also 
recommended agreeing on the development and use of new programmatic tools. They 
specifically mentioned: scheduling a meeting to go over the STEM Pathways Portfolio 
with representatives from each partner organization, developing shared goals as informal 
educators, and providing more opportunities for informal educators to meet. 


Having more time to get to know the other organizations and to talk about what we were 
trying to accomplish and how we were going to assist each other would probably be the 
best thing that we could do. — Steering committee member, STEM Pathways partner 
interviews 
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= Develop concrete goals to accomplish and utilize project management strategies to 
document progress. 


| think that one of the things that makes it hard to speak authoritatively about whether or 
not we're fulfilling goals, whether or not work is being successful, is that it’s not always 
clear throughout the process, what precisely the STEM Pathways are delivering or 
committing to deliver and on what time line. | gave the group some feedback, specifically 
around the need for a more robust planning process and then some really consistent 
project management to back up whatever commitments are being made to make sure 
they’re actually getting built and implemented — MPS leader interviews 


Schedule early and establish communication 


m Partners recommended addressing scheduling challenges. They recommended doing 
so by scheduling visits to partner organizations far in advance, developing and 
adhering to an appropriate sequence for visits to partner organizations for students, 
and identifying one person to coordinate scheduling logistics with the partner 
organizations and schools. 


The suggestions | have... the further out we can plan these things...often these things are 
a month, month and a half in advance, but | would love to know now about next year’s 
schedule. I’ve got programs and my staff have got programs scheduled throughout June 
already. And sort of arbitrarily dropping in or doing a number of Doodle polls to try and 
find something...it’s all just kind of ineffective and right now we should be looking at our 
schedules for next year and trying to get people to either commit to times and holding 
those times so we can get more people participating or something like that. - Steering 
committee member, STEM Pathways partner interviews 


For the fourth grade cohort, | definitely think that summer planning and cohort meetings 
are absolutely necessary because the program takes off so quickly in the fall. | think 
teachers report back to Minneapolis schools on August 17" and | feel like we need to 
have our plan in place, what we're going to do, by the end of August. And then for fifth 
grade, again, early fall planning would be fine, but just kicking things off earlier as a 
cohort. — Implementation team member, STEM Pathways partner interviews 


= Hold an initial kick-off meeting that includes not only the partner organizations, but 
teachers as well. — MPS leader interviews 


= Increase communication between teachers and informal education sites earlier in the 
year and prior to the field trip visits for a more coordinated effort to improve student 
learning. — MPS leader interviews 


Communicate costs and funding 


= Interviewees recommended addressing funding challenges. Their specific 
recommendations included increasing funding to facilitate greater participation from 
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partners who do not have capacity, having each organization commit monetarily to 
the project, and providing transparency regarding what each organization is willing and 
able to do for the project. 


| think in order to make something like this result in more meaningful change and benefit, 
all the partners and Minneapolis schools need to inform us of and commit to what they are 
willing and not willing to do. So we know what we're dealing with. So we know what we're 
working with and can be more planful around that. Who are the decision-makers, who are 
the developers, who are the implementers, and what is each and every person willing and 
able to do and how often. — Steering committee member, STEM Pathways partner 
interviews 


= Communicate costs related to program participation to principals earlier in the year so 
they can plan for STEM Pathways in their budget. —MPS leader interviews 


m= Secure funding for buses and site visits. — MPS leader interviews 


Involve teachers and parents 


= Offer training for the teachers before the school year starts and have a time for them 
to learn from each other about how they incorporate aspects of the field trips into 
their classroom. 


One idea | guess | have is to maybe have all the teachers of STEM Pathways schools talk 
about ways that they incorporate these things in their classroom so that we're not all trying 
to do separate things. Sharing some of those ideas. — MPS leader interviews 


= Provide an opportunity for teachers to give feedback on the experiences at each 
organization. 


Teachers could give some feedback if there was an exit slip to give feedback on what 
went really well this year, what would be another opportunity for learning or what might be 
some ways to make those connections for growth. They could provide more on that. 

— MPS leader interviews 


= Involve/provide opportunities for parents. Provide a list of possible STEM activities 
for parents to do with their children in the community. — MPS leader interviews 


Increase steering committee members involvement 


= Interviewees recommended increasing the capacity, empowerment, accountability, 
and engagement of the steering committee to match that of the implementation team. 


Everyone is trying really hard to provide input when it is asked for. But it’s still kind of they 
are only involved, when they're asked to be involved as opposed to taking initiative. 
— Steering committee member, STEM Pathways partner interviews 
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Research question 3. 
How well does the collaboration function, and how can it be strengthened? 


MPS leaders felt that the collaboration among STEM Pathways partner organizations has 
improved the coordination of STEM experiences for fourth- and fifth-grade students in 
the schools. MPS leaders appreciated how STEM Pathways partner organizations supported 
teachers and communicated with them before and after the field trips or visits. Also, as 
mentioned earlier, MPS leaders felt that STEM Pathways support the district’s curriculum. 


Successes 


Better coordination and alignment of STEM programming in the schools 


Finding: Some MPS leaders felt that participating in STEM Pathways strengthened their 
relationships with the partner organizations and made them more meaningful. 


STEM Pathways allows for a clear balance between the fourth and fifth grade of who's 
going where and then making it more meaningful... It's connected to what they're doing 
and [it's] connected to [what they're doing] during different times of the year. - MPS 
leader interviews 


| think the work we do with them is more coordinated and aligned. — MPS leader 
interviews 


Finding: MPS leaders appreciated how STEM Pathways partner organizations supported 
teachers and communicated with them before and after the field trips or visits. 


The Bakken is helpful because they co-plan with the teachers. [The] staff reached out to 
teachers. A lot of connecting and planning [happened] leading up to the future... The 
Bakken and STARBASE staff have really reached out and have worked a lot with the 
science department. So much work happened behind the scenes. — MPS leader 
interviews 


Strong collaborative work 


Finding: Partners share common goals and value their collaborative work. 


And | think having this shared belief and common understanding and goals and this 
collective brain power is just really critical to our work and success. — Steering committee 
member, STEM Pathways partner interviews 


Strengths of the collaboration are all of us getting together and talking and developing 

these consistent themes and simplifying our messages and getting down to some core 
messages that we're trying to get across. | think that’s valuable. — Steering committee 

member, STEM Pathways partner interviews 


STEM Pathways Evaluation 30 Wilder Research, December 2015 


Challenges 


Varying levels of partner involvement 


Finding: STEM Pathways partners viewed their collaboration positively. However, partner 
organizations had varying levels of engagement in the project. Some partner organizations 
were more able than others to contribute time and resources to STEM Pathways. 


.. everyone Is excited about it and has bought in but there’s not a super good balance of 
how much work different organizations put in toward making stuff happen. And so that can 
create a strange dynamic. And | can imagine down the line, if other organizations aren't 
able to feel sort of like as big of a player that things could start to fall apart. - 
Implementation team member, STEM Pathways partner interviews 


The organizations that don’t have any funding currently, it’s hard for us to fully commit and 
do what we want to do to make this a really great collaboration without having any 
resources behind it. -Steering committee member, STEM Pathways partner interviews 


Research question 4. 
How effective is professional development, and what are its future needs? 


Professional development for STEM Pathways partner educators 


Partners described that they received training about MPS learning targets, state standards 
and curriculum, a writing in science session about science notebooks, and presentations 
from each partner organization about their programming. They appreciated the sharing of 
knowledge among the partners. 


Finding: Sharing knowledge and identifying commonalities between, as well as unique 
talents of, each organization was a beneficial component of the collaboration. 


We get to know more about each other, but we also get to observe and experience other 
teaching styles — other methods of delivering science and engineering concepts. It’s 
always good to see other people in action. Good educators doing their thing in their 
environment, because it always challenges you. Everybody can get complacent in their 
teaching style and their delivery methods. It’s inspiring to see other people and it kind of 
kicks you in the butt and gets you thinking about different ways to present. 

— Implementation team member, STEM Pathways partner interviews 


For future professional development, partners gave the following recommendations: 


= Provide professional development for MPS teachers regarding content knowledge and 
ideas for implementation. 
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= Provide training for informal educators on how to work with English language 
learners (ELL). Provide an opportunity to learn from other informal educators’ 
techniques when working with ELL students. 


= Focus and target the professional development offered so that it is relevant to those 
who attend. Have more single purpose meetings—make sure there are clear goals for 
what needs to be achieved and then follow-up. 


= Provide professional development for the partner organizations to learn about STEM 
integration and how MPS defines it. 


Professional development for classroom teachers 


Similarly, MPS leaders felt that classroom teachers would benefit from the professional 
development with STEM Pathways. 


Finding: Some MPS leaders would like to have professional development offerings for 
teachers who have classrooms involved in STEM Pathways. Others referred to the 
teachers’ involvement in STEM Pathways as a form of professional development and felt 
that teachers were able to benefit from attending STEM Pathways field trips with their 
students. 


| definitely want PD [Professional Development] for our teachers that are implementing... 
So understanding the programming and how to then integrate that into your lessons in 
your classroom. — MPS District leader 


If we had one or two teachers get some more intensive professional development that 
they could bring back to their colleagues or some kind of model where teachers could 
develop their craft a little more with the subject matter. That could follow-up with 
collaboration they're already doing and build capacity for them to do some of this learning 
throughout the day. — MPS Principal 


Wonder if there’s an opportunity for schools and partners to get together for a half day of 
learning or planning around the experiences they're going to have. Set aside some 
planning time. Teachers are always asking for that. - MPS Principal 


! know that the teachers who have gone on these experiences have learned a lot more 
about science, technology, engineering, and math. Their own content knowledge is much 
stronger and that always helps when they're teaching. - MPS Principal 


MPS leaders recommended creating a Professional Learning Community (PLC) around 
STEM. 
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Research question 5. 
What are the core components of the program model and conditions for replication? 


The discussion that follows addresses a portion of this research question regarding the 
conditions necessary or desirable for successful implementation or replication, but does 
not identify the core components of the program’s model. Interviewees identified the 
following key conditions for successful implementation of STEM Pathways: 


= Total buy-in from all partners, the school district, and teachers. 


...buy-in by everybody, but | think teachers are really important. — Implementation team 
member, STEM Pathways partner interviews 


Without someone like [MPS STEM Integrationist] committed to the project it would be 
really hard to make this happen, so that’s really critical. - Steering committee member, 
STEM Pathways partner interviews 


| would also say that it's equally incredible that the partner organizations, most of whom 
have never worked together before or have needed to work together before, have come 
together on this collective work toward this common purpose and all have done so without 
any monetary benefit. That's been pretty amazing because it's no small amount of work 
and no small investment. 

— Steering committee member, STEM Pathways partner interviews 


= Strong leadership and direction from the steering committee. 


| would say commitment of participating partners, maintaining the focus of the goals and 
the deliverables of the Pathways project and having a funding stream that enabled 
someone to spearhead this as part of a main priority or to facilitate the process. 
-Steering committee member, STEM Pathways partner interviews 


= Understanding from teachers about what STEM Pathways is and what the field trips 
involve. 


One of the things that went well, and maybe there was some hesitation at first about how 
it's all going to work, but really having the classroom teachers understand what the project 
is... just connecting with them and helping them understand what STEM Pathways is 
really early on in the year. — Implementation team member, STEM Pathways partner 
interviews 


= Endorsement and advocacy on the part of the district to principals and teachers at 
participating schools. 
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There has to be a strong connection with the expectations of Minneapolis Public Schools 
and this project. If a teacher in the classroom has just received a portfolio, for example, 
from the STEM Pathways liaison and they're left to wonder if this is supported by 
Minneapolis schools, does this count towards anything, does this align or is this an extra 
add-on. That's not good. That needs to be clearer and that needs to be more transparent 
and promoted. So there has to be advocacy there on the part of Minneapolis schools for 
any parts of these pieces that we're trying to accomplish. — Steering committee member, 
STEM Pathways partner interviews 


= Small group work (e.g., fourth and fifth grade cohorts, implementation team, steering 
committee) was described as one of the ways the project was able to get things 
accomplished despite challenges with time and capacity. 


| thought our groups getting together, our small work groups, was very useful. Getting to 
know the other participants was very helpful... Having more time to get to know the other 
organizations and to talk about what we were trying to accomplish and how we were 
going to assist each other would probably be the best thing that we could do. — Steering 
committee member, STEM Pathways partner interviews 


= Collaboration—meaningful input from every partner involved. 


| think that it will be important that there will be equal representation from all of the 
organizations in decision-making. — Steering committee member, STEM Pathways partner 
interviews 


= Consistency of activities—ensuring that each partner consistently implements agreed 
upon programmatic components (e.g., in referencing the STEM Pathways Portfolio). 


| think greater consistency of activities, stronger leadership [or] direction on the part of the 
steering committee... We couldn't rely on just shared messages from the partners 
because we knew that wasn’t happening consistently. And the portfolios weren't being 
implemented consistently. 

— Steering committee member, STEM Pathways partner interviews 


= Funding. 


So | think the biggest gap is funding to the participating organizations. Support funding. 
— Implementation team member, STEM Pathways partner interviews 


STEM Pathways Evaluation 34 Wilder Research, December 2015 


Outcomes 


The first year evaluation assessed short-term outcomes associated with change in student 
perceptions about STEM learning and opportunities, student school outcomes related to 
STEM Pathways long-term outcomes, and overall systems change efforts. 


Student surveys in fall and spring, MPS leader interviews, and student academic 
achievement data were used to address the following questions. 


Research question 6. 


Does the STEM Pathways model enhance the short- and long-term outcomes of populations 
underrepresented in STEM activities? 


Research question 7. 
How well does the model work for specific underserved populations, and in what context? 


Student survey 


Results for the 16 close-ended survey items included in both the fall and spring surveys 
are summarized below. The closed-ended items were a series of statements with the 
response options: agree a lot, mostly agree, agree a little, don’t agree, and don’t know. 
Those who agreed a lot or agreed mostly were considered to be in agreement with an 
item. Results are summarized here by content areas with promising results and opportunities 
for growth in student learning. Results in detail can be found in the full report, STEM 
Pathways Student Survey Results for the 2014-15 School Year (Mueller, 2015). 


Promising results 


STEM interest and application 


Interest in engineering (“I like learning engineering”) increased overall from fall to spring 
(agreement with the statement increased from 62% to 77%). This increase occurred 
primarily among fourth-graders (agreement increased from 57% to 81%; Figure 2). The 
increase appeared to be due to increased knowledge or understanding of what engineering 
is as “don’t know” responses from fourth-graders went down sharply from fall to spring 
(24% to 7%) while “agree a lot” responses went up very strongly (38% to 58%). The 
share of fifth-graders who indicated having interest in engineering did not change 
significantly from fall to spring, with a quite high percentage of the students agreeing 
with the statement at both times (68% in fall and 72% in spring, Figure 2). 
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2. I like learning engineering: Overall results 
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*The change in the percentage of students responding “agree a lot” or “mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 


Perhaps related to this, agreement with the statement, “I think like an engineer to design 
solutions to problems,” increased among fourth-graders from fall to spring (from 50% to 
58%). Boys showed more interest in engineering than girls, but the interest of both groups 
increased from fall to spring. The interest in engineering of all other demographic groups 
examined increased from fall to spring as well. 


Interest in STEM as a whole (“I like leaning STEM”) increased among fourth-graders 
from fall to spring (agreement with the statement increased from 73% to 78%; Figure 3). 
This increase occurred especially among boys (from 73% to 79%) and ELL students 
(from 70% to 78%; Figure 4). 


3. I like learning STEM: Overall results 
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*The change in the percentage of students responding “agree a lot” or “mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 
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4. | like learning STEM: Boys and ELL students 
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*The change in the percentage of students responding “agree a lot” or “mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 


Overall, interest in other STEM subjects (math, science, and technology) did not change 
significantly from fall to spring. These results might still be viewed as favorable because 
over three-quarters of the students already liked learning these subjects at the time of the 
baseline survey in the fall. Despite the lack of change overall, interest in both science and 
technology increased among boys from fall to spring. 


STEM relevance and awareness 


Students’ agreement that STEM knowledge is important to their futures increased from 
fall to spring (from 68% to 77% in agreement; Figure 5). This increase occurred across 
grades and across almost all student demographic groups examined. 


5. STEM knowledge is very important to my future: Overall results 
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*The change in the percentage of students responding “agree a lot” or “mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 
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Overall, students’ awareness of STEM (“I notice STEM in the world around me every 
day”’) also increased from fall to spring (from 54% to 59% agreeing with the statement). 
This increase primarily occurred among fourth graders (from 52% to 61% agreeing with 
the statement; Figure 6) and among students of color (Figure 7). 


6. | notice STEM in the world around me every day: Overall results 
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*The change in the percentage of students responding “agree a lot” or ‘mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 


7. notice STEM in the world around me every day: White students vs. 
students of color 
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*The change in the percentage of students responding “agree a lot” or “mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 


Knowledge of STEM careers 


Knowledge of STEM jobs (“I know about many jobs that use STEM”) increased from 
fall to spring in both fourth and fifth grades (from 59% to 68% agreeing with the 
statement). This increase occurred in most of the student demographic groups examined. 
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Challenges and possible opportunities for growth 


Confidence in STEM abilities 


Confidence in STEM abilities (“I am really good at STEM”) decreased from fall to spring 
(from 58% to 49% agreeing with the statement). Fourth-graders had a bigger decrease 
than fifth-graders, with more fourth-graders being less sure of their STEM abilities in the 
spring (i.e., more responding “don’t know” to the survey item, 17% in fall to 29% in 
spring). Student demographic groups with decreases in their STEM confidence levels 
were girls, low-income students, ELL and non-ELL students, and students of color 
(Figures 8 and 9) 
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*The change in the percentage of students responding “agree a lot” or “mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 


9. lamreally good at STEM: High- vs. low-income students 
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*The change in the percentage of students responding “agree a lot” or “mostly agree” to the item from fall to spring was 
statistically significant (p<.05). 
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Despite the decrease in agreement with the statement, “I am really good at STEM,” 
agreement with the statement, “I would be good at a job that uses STEM,” did not change 
significantly from fall to spring (slightly over 50% agreed at both time points). Girls, low- 
income students, and students of color tended to be less confident that they would be good 
at a STEM-related job compared to their demographic counterparts. 


Participation in STEM activities outside of school 


The proportion of students engaging in frequent STEM-related activities outside of school 
did not change significantly from fall to spring in either fourth or fifth grade. Overall, 39 
percent of the students in the fall survey agreed that they frequently did STEM-related 
activities outside of school and 36 percent in the spring survey agreed they frequently did 
such activities in the spring. The proportion of ELL students participating frequently in 
such activities declined from fall to spring (from 45% to 32%). Figure A2 shows the 
student responses to the open-ended question about the STEM-related activities they did 
outside of the school day in the spring student survey. 


Slightly fewer than half of the students in both the fall and spring agreed that they knew 
about many STEM-related activities outside of school. The only demographic group to 
have an increase in agreement with this item from fall to spring was white students. 


Over 60 percent of the students in both fall and spring agreed that they would like to do 
more STEM-related activities. 


Application of technology 


Applying technology to problem-solving (“I use technology to solve problems”) did not 
change from fall to spring with 56 percent of all the students agreeing with this item at 
both time points. 


Interest in STEM careers 


There was little change from fall to spring in students’ interest in having a job that uses 
STEM when they are older, with slightly over half agreeing with this item at both time 
points. Higher income students were more likely to have an interest in such a job. 


STEM Pathways Evaluation 40 Wilder Research, December 2015 


MPS leader interviews 


In their interviews, MPS leaders indicated that STEM Pathways strengthens student 
STEM skills and provides exposure to STEM careers. MPS leaders recognized that 
STEM Pathways targeted lower-income schools and focused on closing the opportunity 
gap for these students. 


Finding: STEM Pathways expands student access to STEM learning activities and 
promotes career exploration. 


Yeah, | think motivation in middle school especially. | guess even in the upper elementary 
grades. Sometimes kids will say, ‘Why do | need to learn this?’ And now there are very 
practical and obvious things we can point to. Well if you want to have a job in technology, 
make a video game, or do any of those things you need those skills. And | know a lot of the 
field trip experiences show those career pathways. — MPS leader interviews 


Finding: STEM Pathways sets up students for success by strengthening skills around 
science, technology, engineering, and math. It also helps students make connections 
between STEM and everyday things. 


Being able to say, ‘building a bridge is really cool’ or ‘building a skyscraper’ or ‘being a 
scientist that is able to help with finding a cure for something.’ Or, as a mathematician, | know 
all these pieces, but if | want to expand it | can become an architect or if | want to become a 
chemist or any of those things. So it's really about the kids starting to have these connections 
made for them. It’s not just in a white lab coat wearing goggles or a mathematician writing 
problems. It’s really showing the kids, by learning this, this is the bridge that helps you get to 
that next point. - MPS leader interviews 


Finding: STEM Pathways will increase the likelihood that students will take more STEM 
classes in later grades and will pursue careers in STEM. 


So | believe when we get students excited about science, technology, engineering, and math 
in the early years, we're going be a lot more likely to see them pursuing that rigorous 
coursework down the road. And maybe eventually careers in those fields. And so | think 
we're doing them a great service in terms of sparking their interest. Helping them compete 
down the road in the job market and globally. So | think that’s probably the bigger value of it. 
— MPS leader interviews 


Finding: STEM Pathways provides students with good experiences in science and math. 


It's just a really positive experience for kids. I’m hoping that it will have an impact on 
students’ achievement. We are just kind of getting scores in and it's always hard to figure 
out, kind of, what is it that helped our test scores, was it a field trip, was it a project that 
they did? It's hard to say, but | think having positive experiences in science and math is 
always a good thing and will help student learning. - MPS leader interviews 
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Finding: STEM Pathways provides experiences for students who may not be exposed to 
STEM related activities outside of school. 


Bottom line they get the experience and the exposure that as a school with students that 
are 95% in poverty, 20% in shelters, they're not getting the opportunities to go to places 
like this and this may be one of their only opportunities. - MPS leader interviews 


Student academic achievement 


STEM Pathways potential effects on student academic achievement were examined using 
the spring 2015 Minnesota Comprehensive Assessments, Series IIT (MCA-III). We 
consider MCA tests to be long-term indicators of potential effects of the program. 
Significant program effects may be unlikely in the short term, but could emerge after 
multiple years of program exposure. Data were also examined for underrepresented group 
which includes students from low income families, racial/ethnic minorities, and females. 


Overall results 


A series of statistical analyses were carried out to determine whether MCA math, reading 
and science (available for fifth grade only) scores differed significantly between the 
treatment and comparison groups. Findings are reported in Figure 10, with fourth and 
fifth grade findings shown separately. Average (mean) scale scores were reported for 
each MCA test by group as well as the percentage that reached the proficiency level on 
the test (i.e., met or exceeded grade-level standards). 


For the most part, there was little difference in MCA scores between the treatment and 
comparison groups. In fourth grade, average scale scores and proficiency percentages 
were slightly higher in the comparison group than the treatment group. These differences 
were not statistically significant for math scores and proficiency percentages, but they 
were significant for reading scores and proficiency percentages. In fifth grade, the average 
scale scores for math, reading, and science were almost identical for the treatment and 
comparison groups (one-point difference or no difference). Proficiency percentages were 
also very similar across the groups for math, reading, and science (one-two percentage 
point differences). None of the differences between groups for fifth-graders were 
Statistically significant. Note that half or fewer of the students were proficient on each of 
the tests across grades and study groups (ranging from 28% to 50%). Among fifth- 
graders, 28 and 30 percent, respectively, were proficient in science in the treatment and 
comparison groups. 
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10. Achievement tests (MCA-III) results in spring 2015: Treatment and 
comparison groups 


Group 
Treatment Comparison 
4" Grade (N=355) (N=1,067) 
Math Scale score mean® 446 448 
Percent proficient® 45% 50% 
Reading Scale score mean? ¢ 441 444 
Percent proficient®¢ 31% 37% 
Treatment Comparison 
5" Grade (N=331) (N=1,059) 
Math Scale score mean? 543 543 
Percent proficient® 36% 37% 
Reading Scale score mean? 544 545 
Percent proficient® 40% 39% 
Science Scale score mean? 539 540 
Percent proficient® 28% 30% 


aln 4h grade, scale scores range from 409 to 499 in math and 411-490 in reading. 
5 Percent meeting or exceeding standards. 
¢ The difference between the treatment and comparison group Is statistically significant (p<.05). 


qIn 5h grade, scale scores range from 515 to 586 in math, 517-591 in reading, and 501-599 in science. MCA science is only 
administered to fifth grade students. 


Additional comparisons within student demographic categories 


Potential treatment-comparison group differences on the 2015 MCA tests were examined 
separately within the following demographic categories: female, male, eligible for free or 
reduced-price lunch, ineligible for free or reduced-price lunch, ELL, non-ELL, Asian, 
black, Hispanic, and white. (There were too few students to conduct the analyses within 
the American Indian group). No strong patterns of differences emerged within any of 
these demographic categories. Overall, differences tended to be small and most often 
slightly in favor of the comparison group (i.e., slightly higher average MCA scale scores). A 
weak but consistent pattern of MCA score differences in favor of the comparison group 
was found among white students and students ineligible for free or reduced-price lunch 
(higher-income students). Differences in favor of the comparison group, although usually 
small, were consistent across grades four and five and across type of test for these two 
demographic groups. 
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A full description of results and the statistical techniques used in analyzing the student 
academic data is presented in another report: STEM Pathways student academic 
achievement results for the 2014-15 school year (Mueller, 2015). 


Research question 8. 
What impacts does the model have on informal STEM education organizations? 


STEM Pathways partners felt that they have become more familiar with what other 
partner organizations do and can appreciate their similarities and differences. Many 
partners commented on having the realization of a rich STEM community in the Twin 
Cities and that this has helped them feel supported in what they do. Leaders at the partner 
organizations felt impassioned by the staff at the other organizations and enjoyed getting 
to know them, particularly through presentations from participating organizations on their 
programming. Many organizations said they realized that they are not a stand-alone 
organization, but rather, what they have to offer complements the work of the other 
partners. 


Finding: Familiarity with, knowledge of, and appreciation for what other partner 
organizations do and how their work complements the work of other informal STEM 
education organizations has increased substantially. 


I've gotten to know these other informal educators and I’ve enjoyed working with them. 
I've gotten to know them better. It's a nice thing. | know the roles of the different informal 
education programs better, what their focuses are. — Implementation team member, 
STEM Pathways partner interviews 


This past year working with these other institutions and realizing that it is genuinely trying 
to be cooperative. It's really helped me to realize what our place in the whole Twin 
Cities/Minnesota region is, what our niche is, what our role is in the community, and how 
these other organizations are different. So that if a school came to me or a teacher came 
to me and said, ‘oh, we'd really like to do a program on the human body and electricity’ or 
something, | could say, we could do something like that, but you should really go to The 
Bakken. Or, in another case, you might want to go to the Minnesota Zoo. | have a better 
idea of how we're cooperative and can be cooperative rather than being in direct 
competition. — Steering committee member, STEM Pathways partner interviews 
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Research question 9. 
What impacts does the model have on classroom teachers? 


MPS leaders indicated that involvement in STEM Pathways provided benefits to 
classroom teachers, including providing modeling for teachers and supporting them in 
presenting STEM curriculum and career options to students 


Finding: STEM Pathways provides modeling for teachers, inspires them, and increases 
their interest in STEM topics. It gives them new ideas for ways to incorporate STEM 
education into their classroom. 


| think it's helped them...1’m thinking mostly about our science teachers. Science is often 
times a subject in elementary grades that you do when you have time to do it, because 
there’s so much you have to do in terms of literacy and math. This has been a chance for 
teachers to really dig into science and feel the importance of that content and have the 
tools and resources to really teach it in depth. And | think this has really helped that. I’m 
thinking, specifically about some of the vocabulary that is used in all of these experiences 
and then also back in the classroom. | think that's been really great. I've seen teachers 
talk about those field trip experiences in the lessons that they're doing in the labs that 
they’re doing back here. So that's been very helpful. ...Last year we departmentalized so 
there was just one fourth grade teacher who taught all the science, and | think she got 
very good at this. She knew exactly what she needed to teach them, and | think that was 
very helpful with all those experiences. — MPS leader interviews 


Finding: Principals feel that teachers felt supported by STEM Pathways and that it 
helped alleviate some of the pressure around planning and organizing meaningful 
experiences for students. 
Just understanding what standards are being addressed by each partner helps us to 
coordinate student learning, but also helps teachers as they're doing their planning to be 


very purposeful ... there’s an opportunity for these preliminary and post-activities and 
tying it in a meaningful way. — MPS leader interviews 


| think having that support. It's not a ‘here, do this’ program, but the teachers are getting 
the support. There’s follow-up, they're meeting and having check-ins with the coordinators. | 
think that’s important and a positive thing that gives them the support they need and also 
keeps them on the right track. - MPS leader interviews 


Finding: STEM Pathways assists and empowers teachers to make connections to the 
MPS curriculum and broadens their ability to present STEM curriculum, including 
providing stronger foundations for students in math and science to support their 
knowledge in engineering, and present STEM career options to students and career 
options to students. 
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! think the professional development they get by going [on the field trips] with their 
students, but also the ones who participate in the planning group around the STEM 
Pathways, | think will feel more connected to the work at MPS and will help us build our 
institutional knowledge around what quality STEM instruction looks like for elementary 
age students. - MPS leader interviews 


think it also gives teachers a sense of empowerment, to the extent that they are planning 
and leading around STEM Pathways programming, not just experiencing something that’s 
been designed by other people for their benefit without any of their input. - MPS leader 
interviews 


It's an opportunity for our students and our teachers and our 4th and 5th grade to have 
beginning exposure to STEM and to build that catalyst that will get them even more 
interested to seek further information, seek further in professional development and being 
able to integrate it into classroom. So | think the largest piece is making those connections 
across the standards and bringing them in for STEM for what we know about our 21st 
century learners. The importance of having a strong science foundation, a strong math 
foundation, so that all of those come together and continue to support that engineering 
piece. And what we know is that anything that’s hands-on for our students, such as 
STARBASE and really hits that engineering goal, our kids did exceptionally well in terms 
of their participation and their behavior. It showed the gaps in what they may or may not 
know overall in those areas, but it was a piece that when the kids come back they were 
like, ‘can we have our classroom like this?’ It’s beginning to contribute to building that, so 
that is one of the kind of overall goals. ... Building a knowledge base around those key 
disciplines, looking at how you integrate/make those connections so that our kids are 
ready to be 21st century learners. - MPS leader interviews 


Research question 10. 
What are the implications for the field of informal science education? 


MPS leaders had positive views on the implications that STEM Pathways might have for 
STEM education, including in increasing students’ interest in science and preparing them 
for the 21“ Century skills. 


It's really the catalyst to get our students to think about what it means to be a 21% century 
learner and a citizen of the 21° century. —- MPS leader interviews 


! think it’s a model that other organizations in different areas of Minnesota or across the 
nation could really adopt and look at. | think it has a lot of potential in [conveying] what we 
want STEM education to look like. - MPS leader interviews 


Especially with MCAs, a lot of the focus is on reading and math, and science kind of gets 
put to the side. So | think something like this, the STEM Pathways program, keeps the 
sciences, especially the sciences, alive—doing it in the classroom and beyond. Earlier | 
said I've seen a lot more science going on in fourth and fifth [grade] classrooms this year 
than | have in the past. - MPS leader interviews 
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Partners felt that the STEM Pathways model can be used as an example of strong 
collaboration among informal education organizations that can support and improve 
student learning at school. 


Finding: STEM Pathways can serve as an example and model for other informal 
organizations and other communities. 


! would say we're setting ourselves up as a model for other communities throughout the 
country. Hopefully it will have the results that we hope will come out of it. We've kind of 
become the icon, the hallmark, the leader of the pack and could maybe get programs like 
this started in other locations. — Implementation team member, STEM Pathways partner 
interviews 


If we can show that we can have a great impact on student learning by combining our 
efforts and helping students see the connections between these different experiences, | 
think that has a tremendous impact on other institutions in different communities who are 
looking to broaden their impact as well. - Steering committee member, STEM Pathways 
partner interviews 


| also think every community and every school district is different so it needs to be a 
model that can be tweaked to meet the specific needs of the community where it’s being 
implemented. — Steering committee member, STEM Pathways partner interviews 


Finding: STEM Pathways provides a strong example for how informal science education 
can support and improve classroom learning. 


The potential for this type of work is to show that there are actual ways to collaborate and 
to do integrated science education that brings in the humanities, the reading, the social 
studies that schools say that they want, what education says is needed. This is something 
that can do that in a small way, but as a much better example of how it could be done 
than what currently exists. So an implication could be if it worked it could have broad 
impact on what we do and could be a case example for what other people do in the future. 
— Steering committee member, STEM Pathways partner interviews 
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Looking forward 


STEM Pathways partners continue to strengthen their partnership and deliver programming 
to fourth- and fifth-grade students in the same six MPS schools. Fourth-grade students 
from 2014-2015 who are enrolled in one of the STEM Pathways schools in 2015-2016 as 
fifth-graders continue to receive STEM learning experiences from STARBASE and two 
new partners (Minnesota Zoo and Bell Museum). Another cohort of fourth-grade students 
in these schools also start participating in STEM Pathways, receiving programming from 
The Bakken, the Works Museum, and STARBASE. 


As STEM Pathways is well into the third year of the project, partners are focusing on 
strategic planning to determine the next phase of the project. Through this process, 
STEM Pathways partners will consider: 


= How to facilitate and support sustainability of access to a system of high-quality and 
interconnected informal STEM education for youth in collaboration with schools and 
districts. 


= Articulating and strengthening of the local STEM learning ecosystem. 


= Continuing to promote cross-organizational leadership to create and prioritize a 
culture of collaboration that builds authentic connections between organizations, 
people and programs; that articulates shared vision and goals; and utilizes shared 
measurement to evaluate progress toward goals. 


= Continuing to support and even expand the network of informal STEM educators that 
STEM Pathways has created. 


m Strategies for sustainability. 
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Appendix 


Student open-ended responses 


Figures Al — A4 present the results of the open-ended questions from spring student 
survey. The responses to these open-ended questions were coded according to a unique 
coding scheme for each question. Each student’s response was assigned up to five codes. 
The number and proportion of students with the responses are presented for fourth- and 
fifth-graders separately and for all respondents combined. Some students did not provide 
any answers to the open-ended questions. 


Student most frequently described non-structured activities when describing the STEM- 
related activities they did outside of the school day. One in five students mentioned 
activities related to fixing or building things, crafts, cooking or gardening (22%). They 
described using electronic devices, such as computers, phones, and iPads (16%); and 
playing or sports (10%) as STEM-related activities they did outside of the school day. 
They also mentioned doing math (14%) or homework (11%) (Figure A1). 


A1. Please list the STEM-related activities you do outside of the school day 


4" grade 5‘ grade All students 
(N=274) (N=289) (N=563) 
Examples N % N % N % 
Nature/life sciences 7 3% 23 8% 30 5% 
Fixing/building things/crafts/cooking/gardening 62 23% 62 21% 124 22% 
Experimenting/testing 23 8% 16 6% 39 7% 
Using/counting money 4 2% 3 1% 7 1% 
Homework 24 9% 39 14% 63 11% 
ST Math 7 3% 5 2% 12 2% 
Video games 12 A% 9 3% 21 A% 
Use computer/iPad/phone/electronics 36 13% 51 18% 87 16% 
Running/playing/sports/exercise 54 20% 44 15% 98 17% 
Reading 9 3% 7 2% 16 3% 
Coding/programming 5 2% 5 2% 10 2% 
Robotics 11 4% 8 3% 19 3% 
Rockets 3 1% 4 1% 7 1% 
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A1. Please list the STEM-related activities you do outside of the school day 
(continued) 


4‘ grade 5th grade All students 
(N=274) (N=289) (N=563) 
Examples N % N % N % 
Out-of-school programming 
STARBASE 9 3% 6 2% 15 3% 
MN Zoo - = 4 1% 4 1% 
The Bakken 9 3% 9 2% 
The Bell Museum 7 - 1 <1% 1 <1% 
The Works 7 3% - - 7 1% 
GEMS or GISE 8 3% 31 11% 39 7% 
Other After School (e.g., Math Masters, 
Math Squad) 4 2% 10 4% 14 3% 
Restated field, without example 
Science (restated specific field or letter "S") 25 9% 15 5% 40 7% 
Technology (restated specific field or letter "T") 20 7% 21 7% 41 7% 
Engineering (restated specific field or letter "E") 17 6% 10 A% 27 5% 
Math (restated specific field or letter "M") 44 16% 37 13% 81 14% 
STEM (no activity is specified) 5 2% 5 1% 
Other, none, and don't know 
Other-science related 1 <1% 9 3% 10 2% 
Other-engineering related - - 4 1% 4 1% 
Other-math related 6 2% 4 1% 10 2% 
Other 21 8% 19 7% 40 7% 
None/I don't do any 41 15% 34 11% 75 13% 
Total 


Note: — Students can give multiple answers and their responses received multiple codes. Some students did not provide any answers. Small 
numbers of responses for a category (<1% for total students) are not included in the Figure, except for Bell Museum. 
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Through their experiences being a scientist and engineer in a STEM Pathways program, 
students indicated they gained confidence in their abilities overall, gained confidence in 
their STEM skills, enjoyed learning about STEM topics, and became aware of STEM 
career options or of their interest in those careers (Figure A2). 


A2. Tell us one thing you learned about yourself while being a scientist or engineer 
with [organizations] 


4" grade 5" grade All students 
(N=126) (N=100) (N=226) 
N % N % N % 

Build own character/confidence 

| can accomplish something if | improve/try hard/learn 

more/ don’t give up 13 10% 5 5% 18 8% 

| can build/make something/design something 9 7% 5 5% 14 6% 

| am good at something/capable/smart 6 5% 7 7% 13 6% 

| like/am good at working with others/groups 4 3% 7 7% 11 5% 
Gain confidence in STEM skills 

| am good at science 7 6% 6 6% 13 6% 

| am good at engineering 7 6% 6 6% 13 6% 

| am good at mathematics 2 2% 2 2% 4 2% 

| am good at technology - - 5 5% 5 2% 

| am good at STEM overall 5 A% 2 2% 7 3% 

| am good at building/making things/designing things 6 5% 3 3% 9 A% 
Like or enjoy the learning 

| like/love/enjoy it; | thought it was fun/interesting 12 10% 2 2% 14 6% 

| like/enjoyed/interested in STEM or subtopic of STEM 28 22% 13 13% 41 18% 

| like learning about STEM careers 5 4% 2 2% 7 3% 

| like learning about nature/animals - - 5 5% 5 2% 

| like building/fixing/creating things 4 3% 2 2% 6 3% 

| like learning/learning is fun 2 2% 3 3% 5 2% 

| like to solve problems 1 1% 2 2% 3 1% 
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A2. Tell us one thing you learned about yourself while being a scientist or engineer 


with [organizations] (continued) 


4" grade 5" grade All students 
(N=126) (N=100) (N=226) 
N % N % N % 
Interest in STEM careers/know career options 
| can be/I like to be a [STEM career] 1 1% 7 7% 8 4% 
Scientist 3 2% 3 3% 6 3% 
A career that has to do with technology iL 1% 4. 1% 2 1% 
Engineer 13 10% 8 8% 21 9% 
| can have many career options 3 2% - - 3 1% 
Other, none, and don't know 
Other 6 5% 14 14% 20 9% 
None/I don't do any/Nothing 2 2% 5 5% 7 3% 


Note: — Students can give multiple answers and their responses received multiple codes. The organizations listed in the fourth grade survey are The 
Bakken Museum, The Works Museum, or STARBASE and those listed in the fifth grade survey are The Bell Museum, Minnesota Zoo, or STARBASE. 
Small numbers of responses for a category (<1% for total students) are not included in Figure A2. 
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Students described many similarities in their STEM learning experiences across STEM 


Pathways programs. Their most frequent answers included learning about science, 
STEM, and about animals or nature. They also mentioned other learning not related to 


STEM, such to challenge themselves, to respect others, and that learning is fun (Figure 


A3). 


A3. Name one thing that was similar about what you learned at [Organization] and 


what you learned at [Organization] * 


4" grade 5" grade All students 

(N=244) (N=248) (N=492) 

N % N % N % 
Electricity/Energy 17 7% 1 <1% 18 4% 
Building 21 9% 3 1% 24 5% 
Science 45 18% 38 15% 83 17% 
Technology 27 11% 19 8% 46 9% 
Engineering 21 9% 9 A% 30 6% 
Math 3 1% 3 1% 6 1% 
STEM- no specific content/topic 80 33% 62 25% 142 29% 
Other similarity made (not STEM related) 25 10% 28 11% 53 11% 
Other similarity made by topic/content (e.g., problems in the 
world; talk about human discovery; something to do with 
water) 12 5% 9 4% 21 4% 
Robots 2 1% 2 1% 4 1% 
Mars/space/rover 1 <1% 4 2% 5 1% 
Problem solving 3 1% 6 2% 9 2% 
How things work 2 1% 2 1% 4 1% 
STEM jobs 9 4% 14 6% 23 5% 
Extinction/Endangered species - 6 2% 6 1% 
Habitats - - 8 3% 8 2% 
Environment - - 7 3% 7 1% 
Animals/bees/nature - - 48 19% 48 10% 
Survival - - 3 1% 3 1% 
None/I don’t have any 5 2% 9 4% 14 3% 


Note: — Students can give multiple answers and their responses received multiple codes. Small numbers of responses for a category (<1% for total 


students) are not included. 


a Two of the three organizations were selected randomly. For fourth-grade survey, the three organizations are The Bakken Museum, The Works 
Museum, and STARBASE and for the fifth grade survey, they are the Bell Museum, Minnesota Zoo, and STARBASE. 
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Students mentioned that their STEM Pathways experiences will help prepare them for a 
STEM career (57%), makes them smarter (17%), help them to know how to help others 
(17%), help them in school or prepare them for college (5%), and know how to care for 


the environment (1%) (Figure A4). 


A4. What have you learned from the [organizations] that you think will help you now 


or in your future? 


4" grade 
(N=116) 
Good for jobs/learned | can be a [STEM career]/learned 
about being a [STEM career] 68 59% 
Increase knowledge/makes me smarter/learned about 
something 23 20% 
Ways to help others or the world/animals 11 10% 
Helps with school/college 7 6% 
Environmental sustainability/helps me care for environment - - 
Other, none, and don't know 
Other 9 8% 
None/I don't do any 1 <1% 


5" grade 
(N=107) 

59 55% 
14 13% 
26 24% 
4 A% 
2% 
5% 
4 4% 


All students 
(N=223) 
127 57% 
37 17% 
37 17% 
11 5% 
2 1% 
14 6% 
5 2% 


Note: Students can give multiple answers and their responses received multiple codes. The organizations listed in the fourth grade 
survey are The Bakken Museum, The Works Museum, or STARBASE and those listed in the fifth grade survey are the Bell Museum, 


Minnesota Zoo, or STARBASE. 
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